Add drain option to dask-worker
#8752
Conversation
|
Can one of the admins verify this patch? Admins can comment |
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 27 files ±0 27 suites ±0 11h 29m 34s ⏱️ - 10m 9s For more details on these failures, see this check. Results for commit 75e98c3. ± Comparison against base commit 5589049. ♻️ This comment has been updated with latest results. |
|
Found that |
| @click.option( | ||
| "--drain/--no-drain", | ||
| default=False, | ||
| help="Let the worker finish its current work before closing [default: --no-drain]", | ||
| ) |
There was a problem hiding this comment.
I like the idea of having a flag you can set which allows workers to complete current tasks without accepting new ones. It's probably most useful when tasks a very large, which is not best practice, but common enough that we should make QoL improvements for those users.
I'm not sure on the drain terminology. In SLURM it means the same thing as it means here, finish existing work without accepting new work. However, in Kubernetes terminology it means evict all current work and reschedule tasks on another worker. So for Dask users in the Kubernetes community this may be confusing.
Perhaps this option could be made mode explicit like --complete-tasks-on-shutdown. Although that's pretty long, so if you can think of something better than that's great!
Closes #3141
pre-commit run --all-filesNeed some help on adding appropriate tests for this option